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Abstract 

This paper presents a numerical method to calculate the value function for a general 
discounted impulse control problem for piecewise deterministic Markov processes. Our 
approach is based on a quantization technique for the underlying Markov chain defined 
by the post jump location and inter-arrival time. Convergence results are obtained 
and more importantly we are able to give a convergence rate of the algorithm. The 
paper is illustrated by a numerical example. 
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1 Introduction 

We present here a numerical method to compute the value function of an impulse control 
problem for a piecewise deterministic Markov process. Our approach is based on the 
quantization of an underlying discrete-time Markov chain related to the continuous-time 
process and path- adapted time discretization grids. 

Piecewise-deterministic Markov processes (PDMP's) have been introduced in the lit- 
erature by M. Davis [6] as a general class of stochastic hybrid models. PDMP's are a 
family of Markov processes involving deterministic motion punctuated by random jumps. 
The motion of the PDMP includes both continuous and discrete variables {(X(i), T(i))}. 
The hybrid state space (continuous/discrete) is defined as R d x M where M is a countable 
set. The process depends on three local characteristics, namely the flow 4>, the jump rate 
A and the transition measure Q, which specifies the post-jump location. Starting from 
(x,v) 6 M. d x M the motion of the process follows the trajectory (<j) v (x,t),v) until the 
first jump time T\ which occurs either spontaneously in a Poisson-like fashion with rate 
X u (4> u (x,t)) or when the flow (p u (x,t) hits the boundary of the state-space. In either case 
the location of the process at the jump time T\: T(Ti)) = is selected 
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by the transition measure Q v {cf> v {x, Ti), •). Starting from (Z\, yi), we now select the next 
inter-jump time T2 — T\ and postjump location (X(T2), T(T2)) = (^2,2/2)- This gives a 
piecewise deterministic trajectory for {(X(t), T(i))} with jump times and post jump 
locations {(Zfc,2/fc)} which follows the flow (ft between two jumps. A suitable choice of the 
state space and the local characteristics (ft, A, and Q provides stochastic models covering 
a great number of problems of operations research, see [BJ. To simplify notation, there is 
no loss of generality in considering that the state space of the PDMP is taken simply as a 
subset of K rather than a product space M. d x M as described above, see Remark 24.9 in 
|6j for details. 

An impulse control strategy consists in a sequence of single interventions introducing a 
jump of the process at some controller-specified stopping time and moving the process at 
that time to some new point in the state space. Our impulse control problem consists in 
choosing a strategy (if it exists) that minimizes the expected sum of discounted running 
and intervention costs up to infinity, and computing the optimal cost thus achieved. Many 
applied problems fall into this class, such as inventory problems in which a sequence of 
restocking decisions is made, or optimal maintenance of complex systems with components 
subject to failure and repair. 

Impulse control problems of PDMP's in the context of an expected discounted cost have 
been considered in [H |9l [101 [13] . Roughly speaking, in [5] the authors study this impulse 
control problem by using the value improvement approach while in El QUI Q3] the authors 
choose to analyze it by using the variational inequality approach. In [5], the authors 
also consider a numerical procedure. By showing that iteration of the single-jump-or- 
intervention operator generates a sequence of functions converging to the value function of 
the problem, they derive an algorithm to compute an approximation of that value function. 
Their approach is also based on a uniform discretization of the state space similar to the 
one proposed by H. J. Kushner in [12]. In particular, they derive a convergence result for 
the approximation scheme but no estimation of the rate of convergence is given. To the 
best of our knowledge, it is the only paper presenting a computational method for solving 
the impulse control problem for a PDMP in the context of discounted cost. Remark that 
a similar procedure has been applied by O. Costa in [3] to derive a numerical scheme for 
the impulse control problem with a long run average cost. 

Our approach is also based on the iteration of the single-jump-or-intervention operator, 
but we want to derive a convergence rate for our approximation. Our method does not 
rely on a blind discretization of the state space, but on a discretization that depends on 
time and takes into account the random nature of the process. Our approach involves a 
quantization procedure. Roughly speaking, quantization is a technique that approximates 
a continuous state space random variable X by a a random variable X taking only finitely 
many values and such that the difference between X and X is minimal for the L p norm. 
Quantization methods have been developed recently in numerical probability, nonlinear 
filtering or optimal stochastic control with applications in finance, see e.g. [T| \2\ [T4" l [15 ] [TB I 
ITT] and references therein. It has also been successfully used by the authors to compute 
an approximation of the value function and optimal strategy for the optimal stopping 
problem for PDMP's [7J. 

Although the value function of the impulse control problem can be computed by iter- 
ating implicit optimal stopping problems, see [3] Proposition 2 or [BJ Proposition 54.18, 
from a numerical point of view the impulse control is much more difficult to handle than 
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the optimal stopping problem. Indeed, for the optimal stopping problem, the value func- 
tion is computed as the limit of a sequence (v n ) constructed by iterating an operator L. 
This iteration procedure yields an iterative construction of a sequence of random variables 
v n (Z n ) (where (Z n ) is an embedded discrete-time process). This was the keystone of our 
approximation procedure. As regards impulse control, the iterative construction for the 
corresponding random variables does not hold anymore, see Section [4] for details. This 
is mostly due to the fact that not only does the controller choose times to stop the pro- 
cess, but they also choose a new starting point for the process to restart from after each 
intervention. This makes the single-jump-or-intervention operator significantly more com- 
plicated to iterate that the single-jump-or-stop operator used for optimal stopping. We 
manage to overcome this extra difficulty by using two series of quantization grids instead 
of just the one we used for optimal stopping. 

The paper is organized as follows. In Section [2] we give a precise definition of a PDMP 
and state our notation and assumptions. In Section [4j we present the impulse control 
problem and recall the iterative construction of the value function presented in [5J. In 
Section [5j we explain our approximation procedure and prove its convergence with error 
bounds. Finally in Section [6] we present a numerical example. Some technical results are 
postponed to the Appendix. 

2 Definitions and assumptions 

We first give a precise definition of a piecewise deterministic Markov process (PDMP). 
Some general assumptions are presented in the end of this section. Let us introduce 
first some standard notation. Let M be a metric space. B(M) is the set of real- valued, 
bounded, measurable functions defined on M. The Borel cr-field of M is denoted by B(M). 

Let Q be a Markov kernel on (M,B(M)) and w G B(M), Qw(x) = \ w(y)Q(x,dy) for 



x G M. For (a, b) G M 2 , a A b = min(a, b) and a V b = max(a, b). 

Let E be an open subset of Mr, dE its boundary and E its closure. A PDMP is 
determined by its local characteristics ((f), A, Q) where: 

• the flow (f> : M. d x R — > M. d is a one-parameter group of homeomorphisms: <fi is continuous, 
4>(-,t) is an homeomorphism for each t G K satisfying (ft(-, t + s) = (j)(4>(-, s),t). 
For all x in E, let us denote 



From these characteristics, it can be shown [61 p. 62-66] that there exists a filtered prob- 
ability space (fL,J r ,{J't},{Px}xeE) such that the motion of the process {X(t)} starting 
from a point x G E may be constructed as follows. Take a random variable T\ such that 



ill 



t*{x) = inf{t > : <j>[x,t) G dE] 



with the convention inf = oo. 

• the jump rate A : E — > M + is assumed to be a measurable function. 

• Q is a Markov kernel on (E,B(E)) satisfying the following property: 



(Vx G E), Q{x, E - {x}) = 1. 




for t < t*(x) 
for t > t*(x) 
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where for x G E and t G [0, t*(a;)] 

A(x, t) = [ \((f)(x, s))ds. 
Jo 

If Ti generated according to the above probability is equal to infinity, then for t G R+, 
X(i) = cf)(x,t). Otherwise select independently an valued random variable (labelled Z\) 
having distribution Q(<j)(x, Ti), •), namely V X (Z\ G A) = Q(<f)(x, Ti), A) for any A G B(E). 
The trajectory of {X(i)} starting at x, for t < Tl , is given by 

U(*,t) fort<T 1; 
[Zi fort = Ti. 

Starting from X(T\) = Z±, we now select the next inter-jump time T2 — Ti and post-jump 
location X(T2) = Z2 is a similar way. 

This gives a strong Markov process {X(t)} with jump times {Tfc} fceN (where To = 0). 
Associated to {X(t)}, there exists a discrete time process (@n)„ eN defined by O n = 
(Z n , S n ) with Z n = X(T n ) and S n = T n — T n _i for n > 1 and S"o = 0. Clearly, the process 
(@n)neN is a Markov chain, and it is the only source of randomness of the process. 

We define the following space of functions continuous along the flow with limit towards 
the boundary: 

C = {w G B(T') : w{4>(x, •)) : [0, t*(x)) i->- R is continuous for each x G E 
and whenever t*(x) < 00 the limit lim w(<p(x,t)) exists}. 

t— H*(x) 

For w G C, we define w((f>(x, t*(x))) by the limit lim w((f)(x,t)) (note that the limit 

t— >t*(x) 

exists by assumption). Let us introduce L as the set of functions w G C satisfying the 
following properties: 

1. there exists [w] 1 G M+ such that for any (x,y) G -B 2 , n G [0,t*(x) At*(y)], one has 

\w(4>(x,u)) - w{4>(y,u))\ < [wj^x - y\, 

2. there exists [w] 2 G M+ such that for any x E E, and (t, s) G [0, t*(x)] 2 , one has 

\w(4>(x,t)) - w(<p(x, s))| < [iy] 2 |t - s|, 

3. there exists [w\ G M+ such that for any (x, y) G E 2 , one has 

|u;(0(x,t*(x))) - w{4>(y,t*(y)))\ < [w]Jx - y\. 

In the sequel, for any function w in C, we denote by C w its bound: 

C w = sup |w(a;)|. 

The following assumptions will be in force throughout. 
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Assumption 2.1 The jump rate A is bounded and there exists [A], G M+ suc/i that for 
any (x,y) €E 2 ,u€ [0,t*(x) A t*(y)[, 

\\((f)(x,u)) - \(4>(y,u))\ < [A] j | a; - y|. 

Assumption 2.2 T/ie exi£ iime i* is bounded and Lipschitz- continuous on E. 

Assumption 2.3 The Markov kernel Q is Lipschitz in the following sense: there exists 
G such that for any function idGL the following two conditions are satisfied: 

1. for any (x,y) G E 2 , u G [0,t*(x) At*(y)], one has 

\Qw(cf)(x,u)) - Qw{(j}(y,u))\ < [Q] [wj^x - y\, 

2. for any (x,y) G E 2 , one has 

\Qw(4>(x,f(x))) - Qw(<t>(y,t*(y)))\ < [Q] Hjx-y\. 

3 Quantization 

The aim of this section is to describe the quantization procedure for a random variable 

and to recall some important properties that will be used in the sequel. There exists 

an extensive literature on quantization methods for random variables and processes. We 

do not pretend to present here an exhaustive panorama of these methods. However, the 

interested reader may for instance, consult the following works [HI I17j and references 

therein. Consider X an M 9 - valued random variable such that \\X\\ < oo where \\X\\ 

ii lip ii lip 

denotes the L p -nom of X: \\X\\ p = (jE[|X| p ]) . 

Let K be a fixed integer, the optimal L p -quantization of the random variable X consists 
in finding the best possible L p -approximation of X by a random vector X taking at most 
N values: X G {x 1 , . . . ,x K }. This procedure consists in the following two steps: 

1. Find a finite weighted grid r CP with T = {x 1 , . . . , x K }. 

2. Set X = X r where X r = pr{X) with pr denotes the closest neighbour projection 

on r. 

The asymptotic properties of the L p -quantization are given by the following result, see 
e.g. [II]. 

Theorem 3.1 7/E[|X| p+ ' 7 ] < +oo for some rj > then one has 

lim K p ' q min \\X - ^W, = J PQ [ \h\ q ^ q+p \u)du, 
K^oo \T\<K p J 

where the law of X is Px(du) = h(u)X q (du) + v with v _L Ad, J P) d a, constant and X q the 
Lebesgue measure in M q . 

Remark that X needs to have finite moments up to the order p+rj to ensure the above con- 
vergence. There exists a similar procedure for the optimal quantization of a Markov chain 
{^fcjfceN- There are two approaches to provide the quantized approximation of a Markov 
chain. The first one, based on the quantization at each time k of the random variable 
Xk is called the marginal quantization. The second one that enhances the preservation of 
the Markov property is called Markovian quantization. Remark that for the latter, the 
quantized Markov process is not homogeneous. These two methods are described in details 
in |17[ section 3]. In this work, we used the marginal quantization approach for simplicity 
reasons. 
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4 Impulse control problem 



The formal probabilistic apparatus necessary to precisely define the impulse control prob- 
lem is rather cumbersome, and will not be used in the sequel, therefore, for the sake of 
simplicity, we only present a rough description of the problem. The interested reader is 
referred to [5] for a rigorous definition. 

A strategy S = (r n , R n )n>i is a sequence of non-anticipative intervention times (r n ) n >i 
and non-anticipative E'-valued random variables {R n )n>i on a measurable space (0, J-). 
Between the intervention times T{ and Tj+i, the motion of the system is determined by the 
PDMP {X(t)} starting from R{. If an intervention takes place at x G E, then the set of 
admissible points where the decision-maker can send the system to is denoted by U C E. 
We suppose that the control set U is finite and does not depend on x. The cardinal of the 
set U is denoted by u: 

U= {y i : 1 <i <u}. 

The strategy S induces a family of probability measures P% , x G E, on (f2, F). We define 
the class § of admissible strategies as the strategies S which satisfy Too = oo P^-a.s. for 
all x £ E. 

Associated to the strategy S, we define the following discounted cost for a process 
starting at x G E 



J b (x) = E* 



/■OO :v 

/ e- as f(Y s )ds + Y,e- an <Y n ,Y r+ 



where E% is the expectation with respect to P% and {Y{\ is the process with interven- 
tions. The function / then corresponds to the running cost and c(x, y) corresponds to the 
intervention cost of moving the process from x to y, a is a positive discount factor. We 
make the following assumption on the cost functions. 

Assumption 4.1 f is a positive function in L. 

Assumption 4.2 The function c is continuous on E x U and there exist [c^ G M+, 
[c] 2 G M+ and [c] # G M+ such that 

1. for any (x,y) eE 2 ,u£ [0, t*(x) A t*(y)], 

max|c(</>(x,n),z) - c((j)(y,u), z)\ < [c] \x-y\, 

zgU 1 

2. for any x G E, and (t,s) G [0,t*(x)] 2 , 

max |c((/>(x, t), z) — c(4>(x, s), z)\ < [c] 2 \t — s|, 

3. /or any (x,y) G i? 2 , 

max|c(0(x,t*(x)),z) - c(>(y, z) | < [c]Jx-y|, 

4- for any (x, y) G E x U, < cq < c(x, y) < C c , 
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5. for any (x, y, z) E E x U x U, 



c(x,y) + c(y,z) > c(x,z). 

The last assumption implies that the cost of taking two or more interventions instanta- 
neously will not be lower than taking a single intervention. Finally, the value function for 
the discounted infinite horizon impulse control problem is defined for all x in E by 

V(x) = inf J s (x). 

Associated to this impulse control problem, we define the following operators. For x E E, 
t > 0, (v,w) E C 2 , set 



F(x,t) 
Hv(x,t) 

Iw(x, t) 



tAt*(x) 



-as—A(x,s) 



f(<j){x,s))ds, 



E 



- atM *^~ A ^ tM "^v(^(x,tAt*(x))) 

-^(^o)) w ^ (Zo)tAr(Zo)) ) 1{Si ^ At , (Zo)} 



Z = x 



E 



tAt*(x) 



e" QSl U ;(Z 1 )l {5l<Mi « (Zo)} 



-as— A(x,s) 



\Qw{6[x, s))ds, 



Z n = x 



Finally for notational convenience, let us introduce for (v, w) E C 2 , x E E and t > 0. 

J(v,w)(x,t) = F(x,t) + Hv(x,t) + Iw(x,t), 

Kw(x) = F(x,t*(x)) + HQw(x,t*(x)) + Iw(x,t*(x)), 

It is easy to show that for all n E N 



Kv(x) 
J(v, w)(x, t) 



E 
E 
+e 



F(Z n ,f(Z n ))+e- aS ^v(Z n+1 )\Z n = x 



F(Z n ,t) + e 



n+l)*-{S n+ i<tAt*(Z n )} 



-at/\t*{Z n ) 



V 



(Z n ,t A t*(Z n ))l{ 5n+1 > iAt *( Zn )}|Z n = x 



(4.1) 



(4.2) 



Note that these operators involve the original non controlled process {X(t)} and only 
depend on the underlying Markov chain (O n ) = (Z n ,S n ). The equalities above are valid 
for all n because (B n ) is an homogeneous Markov chain. Finally, for (v, w) E C 2 , ip defined 
on U and x E E, set 

M(p{x) = inf {c(x,y) + <p(y)}, 
yen 

L(v,w)(x) = inf J(v, w)(x, t) A Kw(x), 

Cw(x) = L(Mw,w)(x). 

As explained in [SJ, operator £ applied to w is the value function of the single-jump-or- 
intervention problem with cost function w and the value function V can be computed by 
iterating C. More precisely, let h be the cost associated to the no-impulse strategy: 



h(x) 



E r 



e- as f(X s )ds 



for all x E E. Then we recall proposition 4 of [5]. 
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Proposition 4.3 Assume that g is in L and g > h. Define Vg = g and V 9 +1 = C(V 9 ), 
for all n> 0. Then for all x G E 

V(x) = lim V 9 (x). 

n— >oo 

As pointed out in [5], if one chooses exactly g = h, then corresponds to the value 
function of the impulse problem where only n jumps plus interventions are allowed, and 
after that, there are no further interventions. 

Remark 4.4 Note that operator L is quite similar to the operator used in optimal stop- 
ping, see e.g. Q [7^. However, the iteration procedure here does not rely on L but on 
C The difference between operators L and C comes from the operator M that chooses 
optimally the next starting point. This is one of the main technical differences between ap- 
proximating the value functions of an optimal stopping and impulse problems, and it makes 
the approximation scheme significantly more difficult, as explained in the next section. 



5 Approximation of the value function 



From now on, we assume that the distribution of A(0) is given by 5 XQ for some fixed 
point xq in the state space E. We also choose a function g in L satisfying g > h. Our 
approximation of the value function at xq is based on Proposition 4.3 Following the 



approach proposed by M. Davis and O. Costa in [5], we suppose now that we have selected 
a suitable index N such that V(:eo) — ^n( x o) is small enough see the example in section 
[6j We turn to the approximation of Vfj(xo) which is the main object of this paper. In 
all generality, finding an index N such that V(xq) — V^(xo) is below a prescribed level is 
a very difficult problem to solve. However, in particular cases one can hope to be able 
to evaluate the distance between V{x$) and V 9 n {xq). As suggested by M. Davis and O. 
Costa in [5], a value of N can be chosen by calculating Vn(xo) for different values of n and 
stopping when the difference between two consecutive values is small enough. Our results 
of convergence are derived for a fixed but arbitrary N. 

Recall that if Vo = h, then corresponds to the value function of the impulse problem 
where only N jumps plus interventions are allowed. This is an interesting problem to be 
solved in itself. For notational convenience, we will change our notation in the sequel and 



reverse the indices for the sequence (V ; 



n)0<n<N- 



Set 



V N 



g = 

CVr, 



V 9 



V 9 

K iV-n' 



for all < n < N. 



As explained in the introduction, the keystone of the approximation procedure for op- 



timal stopping in [7] is that the analogue of Proposition 4.3 yields a recursive construction 
of the random variables v n (Z n ). Unfortunately, this key and important property does not 
hold anymore here. Indeed, one has: 



v n (Z n ) 



£-v n+ \(z n 



inf E 



F(Z n ,t) + e- aS ^v n+1 (Z n+1 )l {Sn+1 



{S n+ i<tAt*(Z n )} 



+e -«*Atn^) M ^ +1 (</,(z ri ,tAt*(z n ))i {5n+1 > tAt * (Zn)} |z n 



AE 



F(Z n ,t*(Z n )) + e- aS ^v n+l (Z n+1 )\Z, 
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And Mv n+ i((j)(Z n ,t A t*(Z n )) cannot be written as a function of v n+ i{Z n+ i). Hence, we 
have no recursive construction of the random variables v n (Z n ) and we cannot apply the 
same procedure that we used for optimal stopping. Thus, we propose a new procedure 
to evaluate Mv n +i(4>(Z n ,t A t*(Z n )) separately from the main computation of the value 
function. 

Note that for all < n < N, to compute Mu n+ i at any point, one actually only needs 
to evaluate the value functions v n +i at the points of the control grid U. We propose again 
a recursive computation based on the Markov chain (Z n , S n ) but with a different starting 
point. Set Zq = y G U and Sq = 0. We denote by (Zn, Sn) the Markov chain starting 
from this point (y,0). One clearly knows vn = g on U. Now suppose we have computed 
all the v n on U for k + 1 < n < N. Therefore, all functions Mv n are known everywhere. 
We can then propose the following recursive computation to evaluate v k at y E U: 

v N{z y N _ k ) = g(z y N _ k ) , 

v k+n (ZK) = C(Mv k+n+ i,v k+n+1 )(Z^), for all < n < N- k - 1. 

This way, one obtains v k (ZQ) that exactly equals v k (y). Note that, since the functions 
Mv k+n are known, this provides a tractable recurrence relation on the random variables 
v k+n {Z y k ). 

Remark 5.1 Note that this procedure requires the knowledge of function g for all the 
random variables (Zn) n <N-i defined for the different starting points y E U. This is why, 
in general, we are not able to use the no-impulse cost function h. Indeed, it is hard to 
compute this function, especially if we need to know it everywhere on the state space. The 
most practical solution is to take g equal to a upper bound of h, and therefore constant. 



There is yet another new difficulty hidden in the recurrence relation (5.1) above as 
regards its discretization. Indeed, to compute v n (y), one needs first to compute all the 
v k+n {Zn) with 1 < n < N — k, and to compute v k+ \(y) for instance, one has already 
computed all the ffc +n (Z^_ 1 ) for 2 < n < N — k. Unfortunately, one cannot re-use the 
values of f / t +n (Z^_ 1 ) to compute that of v k+n {Zn), so the computation has to be started 
all over again each time, and one has to be very careful in the design of the approximation 
scheme. However, all these computations can be done with the same discretization grids 



for (Zn,S'n), so that our procedure is still reasonably fast, see section 5.2 for details, and 
figure [BTT] for a graphical illustration of our procedure. 



Remark 5.2 The recursive procedure (5.1) is triangular in the sense that one needs to 
compute all the v k+n (Zn) for < k < N and < n < N — k. 

Our approximation procedure is in three steps, as explained in the following sections. 
The first step consists in replacing the continuous minimization in the definition of operator 
L by a discrete-time minimization, on path adapted grids. The second step is specific to the 
impulse problem, and is due to the operator M as explained in details above. The second 
step hence consists in carefully approximating the value functions v n on the control grid U. 
The last step will then be similar to the approximation of the optimal stopping problem 
and will consist in approximating the value functions at the points of the quantization 
grids of the no impulse process. 
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5.1 Time discretization 

We define the path-adapted discretization grids as follows. 

Definition 5.3 For z G E, set A(z) e]Q,t*(z)[. Define n(z) = int(^^) — l, where int(x) 
denotes the greatest integer smaller than or equal to x. The set of points (ii)ie{o,....n(z)} with 
ti = iA(z) is denoted by G(z). This is the grid associated to the time interval [0,t*(z)]. 

Remark 5.4 It is important to note that, for all z G E, not only one has t*(z) ^ G{z), 
but also maxG(z) = i n ( z ) < t*(z) — A(z). This property is crucial for the sequel. 

We propose the following approximation of operator L, where the continuous minimization 
is replaced by a discrete-time minimization on the path-adapted grids. 

Definition 5.5 For (v,w) G L 2 and x G E, set 

L d (v,w)(x) = min J(v , w)(x, t) A Kw(x). 

Now we compute the error induced by the replacement of the continuous minimization by 
the discrete one. 

Lemma 5.6 Let (v, to) £ L 2 . Then for all x G E, 

inf J(v,w)(x,t) - min J(v,w)(x, s)\ < (c f + C w C x + M, + C v (C x + aj) A(x). 
1 t<t*(x) seG(x) 1 V L i£ J 

Proof: We have 

inf J(v, w)(x, t) — min J(v, w)(x, s)\ = min J(v, w)(x, s) — inf J(v,w)(x,t). 

1 t<t*(x) s&G(x) 1 s£G(x) t<t*(x) 

Clearly, there exists i G [0,t*(x)] such that inf J(v,w)(x,t) = J(v,w)(x,i). Moreover, 

t<t*(x) 

there exists < i < n(x) such that t G (with t n ( x ) + i = t*(x)). Consequently, 

Lemma A. 5 yields 

inf J(v, w)(x, t) — min J(v,w)(x, s)\ < J(v,w)(x,U) — J(v,w)(x,t) 

t<t*(x) s£G(x) 



< [C f + C w Cx+ [v] 2 + C v (C x + a))\t-ti\. 

implying the result. □ 

Lemma 5.7 Let (v, w) G L 2 be nonnegative functions. Then for all x G E, 

\L(v,w)(x)- L d (v,w)(x)\ < (Cf + C W C\ + [v] 2 + C V (C\ + a)jA(x). 

Proof: Since the functions v and w are nonnegative, it follows from the definition of L 
and L d that 

\L(v , w)(x) — L d (v, w)(x) I < I inf J(v,w)(x,t) — min J(v,w)(x, s)\. 

t<t*(x) seG(x) 

Now in view of the previous lemma, one obtains the result. □ 
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5.2 Approximation of the value functions on the control grid U 

We now need to introduce the quantized approximations of the underlying Markov chains 
(@n)- More precisely, we need several approximations at this stage, one for each starting 
point y in the control set U. Recall that U = {y l , 1 < i < u}. For all 1 < % < 
u, let (Z^, S'^)o<n<A r -i be the Markov chain (Z n , SVi)o<n<7V-i with starting point Zq = 
y 1 , So = 0, and let (Z^, S^)o<n<iV-i be the quantized approximation of the sequence 
(Z^, S'^)o<n<Af-i, see Section^ The quantization algorithm provides us with a finite grid 
r^ C E x R + at each time < n < N — las well as weights for each point of the grid 
and transition probabilities from one grid to the next one, see e.g. [U [T^l [T7] for details. 
Set p > 1 such that Q n has finite moments at least up to the order p + e for some positive 
e and let p l n be the closest-neighbour projection from E x R + onto (for the distance 
of norm p; if there are several equally close neighbours, pick the one with the smallest 
index). Then the quantization of Q l n conditionally to Zq = y l is defined by 



We will also denote T]l Z the projection of T]^ on E and Tn S the projection of Tn & on M.+. 

Although (Z^jSn) is a Markov chain, its quantized approximation is usually not a 
Markov chain. It can be turned into a Markov chain by slightly changing the ponderations 
in the grids, see [16] . but this Markov chain will not be homogeneous in any case. Therefore, 
the following quantized approximations of operators H, I, K, J and L d depend on both 
indices n and i. 

Definition 5.8 For v G L 2 , w defined on r^ 1; x E E, < n < N — 1, 1 < i < u and 

z 6 T % n Z , consider 



H l n+1 v(z,t) 
T^ +1 w(z,t) 
K +1 w{z) 



Jn+l{v,w){z,t) 



L t ,f hl (v,w)(z) 



-- E 
= E 
= E 
= E 
+ E 



-a(tAt*(Z*))_ 



e- aS «+^(^ +1 )l { 5; +1<tAf *(4) } 



,(^,tAf(z;)))i { j, +i > (Ai4( g, )} 

Z i ■ 

F(Zlt*(Z l n )) + e- aS ^w(Z l n+ i)\Zn 
F(Zlt) + e- a§l n +lw (zl +1 )l 



L {S* <tAt*(Z»)}l 



-a(*A**(Z*)) 



v (4>(Zn,t A t* (Z l n )))l >tM *(zi)\\Zn — Z 



minJ} l +l(v,w)(z,t)AK t n+1 w(z). 



Our approximation scheme goes backwards in time, in as much as it is initialized with 



computing vn at the points of the last quantization grids T^f, then vn-i is computed on 



Tjv-i and so on. 



Definition 5.9 Set v^iy 1 ) = g{y l ) for 1 < i < u. Then, for 1 < k < N — l and 1 < i < u, 

set v k {y l ) = v% (y l ), where 



Vpj 0) 



g(z), 



zer 



AT-fe' 



J k+n- 



i(*) = Lii d (Mv k+n ,v^ k+n )(z), 



zer 



i,Z 
n—l' 



n e {l,...,JV- fc}. 



See figure 5.1 for a graphical illustration of this numerical procedure. 
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Remark 5.10 Note the use of both v k+n and v^ +n in the scheme above. This is due to 
the fact that we have to reset all our calculations for each value function v k and cannot use 
the calculations made for e.g. v k +i because the value functions are evaluated at different 
points, and are approximated with different discrete operators. This is mostly because the 
quantized process (Z^,S^) is not an homogeneous Markov chain. 

We can now state our first result on the convergence rate of this approximation. 

Theorem 5.11 For all 1 < k < N — 1, < n < N — k — 1 and 1 < i < d, suppose that 
A(z) for z G rjj is such that 



d^\\Z % „ — Z^Wp + dp 5£j_i — 5, 



J n+1 



< min {A(z)}. 



d 3 ze rti z 



Then we have 



< 



; fc+r 

+^fc,rall^n ~~ ZnWp + 2[ffc+n+l] ||^n+l ~~ ^V4-l|L + C/H^n+l ~~ ^n+l\ 



\v k+n+1 (Z l n+1 ) - vl +n+l {Z l n+l )\\ + max \v k+n+1 (y) - v k+n+1 (y)\ 



+4 n ||A(4)|| p + +2^3( d 4|| Z , _ zi j p + d 5||^ +i _ ^ +1 y , 



with 



l k,n 



[Q] [v k+n+1 ] „ + 2E 3 \ V \c c {E x + a [t*] ) + 2( [c] ± + [c] 2 [f] ) j 
C C 

+ [vk+n] + [Q] [Vk+n+l],— + — (#1 + E 2 ), 

4,n = C/ + C Vk+n+1 C x +[c] 2 + (C c + C Vk+n+1 )(C x + a), 



d 3 = ( 2 -^ + c c )c x , 

a 

d* = ^(l+[t*)) + C c [t*) 
a 

d 5 = ifl^ + C, 
\ a 

Remark 5.12 Recall that vn = v 1 ^ = vn = g. Hence, one has 

\\vN(Z l N _ k ) -v% k (Z l N _ k )\\ p < [g]\\Z l N - Z l N \\ p and max \v N (y) - v N (y)\ = 0. 

In addition, the quantization error \\Q l n — ® l n \\p goes to zero as the number of points in the 
grids goes to infinity, see e.g. \1J$ - Therefore, according to Definition 5.9 and by using an 



induction procedure max^u \ v k (y) — v k (y)\ can be made arbitrarily small by an adequate 
choice of the discretization parameters. From a theoretical point of view, the error can be 



calculated by iterating the result of Theorem 5.11. However, this result is not presented 



here because it would lead to an intricate expression. From a numerical point of view, a 
computer can easily estimate this error as shown in the example of section^ 
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The proof is going to be detailed in the following sections. We first split the error into 
four terms. For &U.1 < k < N — 1, < n < N — k — 1 and 1 < i < d, we have 



3=1 



where 



\\v k+n (K) ~ v k+n(K)\\p, 

\\L(Mv k+n+1 ,v k+n+1 )(Zi) - L d (Mv k+n+1 

+n+li v k+n+l +n+lj v k+n+l )(K)\\p> 



Hi^CAfWfc+n+i.Ufc+n+i)^) - ^+i(Mv fc+n+ i,^ n+1 )(Z n ,np. 
The first two terms are easy enough to handle thanks to Corollary |A.12 and lemma 5.7 
Lemma 5.13 A upper bound for T\ is 



\\v k+n (zi)-v k+n (zi)\\ p <[v k+n }\\zi 



J n\\V 



Lemma 5.14 A upper bound for T2 is 

\\L(Mv k+n+1 ,v k+n+1 )(Z l n ) - L d (Mv k+n+1 ,v k+n+1 )(Z l n 
< (Cf + C Vk+n+1 C\ + [ c ] 2 + (C c + a fc+n+1 )(C A + a)) ||A(Z;)|| 



The fourth term is also easy enough to deal with as it is a mere comparison of two 
finite weighted sums. 

Lemma 5.15 A upper bound for T\ is 

+n+li v k+n+l )( z n) ~ Ll +l {Mv k+n+u vl +n+1 ){Z l n )\\ p 
< [v k +n+l\ \\ Z n+l - Z n+l\\ p + \\ v k+n+l( Z n+l) - Vk+n+l ( Z n+l) \\ p 

+ max I v k+n+1 (y) - v k+n+1 (y) I . 
yev 

Proof: We clearly have 

\\L d n+1 (Mv k )(Z* n ) - L^ +1 (Mv k+n+u v^ n+1 )(Z^\\ 

< 



ma ? \ J n+l( Mv k+n+l,V k+n +l)( Z n,t) ~ J* +1 (Mv k+n+1 , «£* +1 ) (Z\, t) \ 

teG(z*) 



V 



K l n+l v k+n+ i(Z l n ) - K l n+l vl k +n+1 {Z l n ) 



< ||E[ Ufc+n+1 (^ +1 )-^ n+1 (z; +1 )|^ 



+ 



E 



Mv k+n+l {<p(Zl t A f(Z* n ))) - Mv k+n+1 (0(2* , t A i* (Z;))) 



Z' 



< 



\ v k+n+l( Z n+l) ~ v k+n+l( Z n+i)\\ p + || u fc+n+l(^n+l) ~~ v k + n +l ( Z n+l) 



+ max \v k +n+i(y) ~ v k +n+i(y)\ 

yev 



showing the result. 



□ 
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We now turn to the third term. This is the key step of the error evaluation, because 
on the one hand, this is where we deal with the indicator functions. The main idea is 
that although they are not continuous, we prove in the following two lemmas that the set 
where the discontinuity actually occurs is of small enough probability. This is also where 
our special choice of time discretization grids is crucial. On the other hand, we use here 
the specific properties of quantization. 

Lemma 5.16 For all 1 < i < d, n G {0, . . . , N — 1} and < T] < min {A(z)}, 



IV(z*)<f(^)-ijllp - 



V 



Proof: By using the Chebychev's inequality, one clearly has 



E 



I 1 *- (z*)<f(^)-J P 



P t*(4)<t*(Z»)-77 



< P (\e{Z i n )-t*{Z i n )\>r 1 < 



r+*ipn 7 % 7 % hp 



showing the result. 



□ 



Lemma 5.17 For all 1 < i < d, n G {0, . . . , N — 1} and < rj < min {A(z)} ; 



zer 



i,Z 



max E[|l R+i<sAt , (zi)} - 1 { ^ <Mt . ( ^ )} l|^] 

2 ~ 2[t*} ■ ~- 



Proof: Set < rj < min {A(z)} and s G G(Z l n ). By definition of the grid G(Z l n ) and r/, 



one has s + 77 < t*{Z l n ), see Remark 
written as 



5.4 



Thus, the difference of indicator functions can be 



1 1 {S l n+1 <sM* (JS* )} 1 {S i n+1 <sM* (Z* )} I 
" l 1 R+ 1 <^*(Zi)} - 1 {S« +1 <»Af (^)}l 



{t*(Zi)<t*(Zi)-2 } + 1 {r(Z*)>t*(Zji)-S} 



^ V(^)<^)-f} + 1 {**(^)> S +I}I 1 {^ +1 <4 1 {s; +1 <,}l 

^ 1 { i *(Z*)<t*(Zi)-|} + 1 {|5; +1 -^ +1 |>|} + 1 {f(Z! l )>s+%} 1 {\S' n+1 ~s\<% }• 



This yields 



max E[|l {S i +i<sAt , (Z i )} - l {g , <sAt * { zi)}W Z n] 



< 1 



{**(Z«)<**(Z*)-§}llp 



+ H 1 {l^ +1 -§; +1 l>l}ll P + II s 6 1 g^ ) E [ 1 {**(^)>«+I> 1 {l^ +1 - S |<|}l^n] || p - (5-2) 
On the one hand, Chebychev's inequality gives 



P(K +1 -^ +1 |>^)< 



2P S 



n+1 °n+l 



(5.3) 
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On the other hand, one has 



E [ 1 {t*(^)>s+I) 1 {|Si +1 - s |<|} 



rL\\Z l „ 



E [ 1 {t*(^)>s+i} 1 {«-i<S* +1 < S +2}l^n] 



-{t*(Zi)>s+l} 



s+-. 



\(<j>(Z l n ,u))du 



7' 



<T]Cx. 



Combining Lemma 5.16 and equations (5.2)-(5.4), the result follows. 



(5.4) 
□ 



We now look up the error made in replacing K by K'^ +1 . This is where we use the 
specific properties of quantization. 

Lemma 5.18 For all 1 < i < d, k G {1, . . . , N — 1} and n G {1, . . . , N — k}, one has 

Kv k+n+ i(Z l n ) - k l n+1 v k+n+ i{Z l n ) < C f \\S l n+1 - S l n+1 \\ p + [wfc+„ + i] \\Z l n+1 - Z l n+1 \\ p 



+ \ [Q] [ Ufc+n+1 ] 1 ^ + [Q] [v k+n+1 ]^ + < ^(e 1 + E 2 ) + 2E 3 \\\Zi - Z%. 



a 



Proof: We have 

Kv k+n +l{Zn) ~ K l n+l V k+n+ i{Zl. 

Kv^xiZl) -E[Kv k+n+1 (Zi)\Zi] 



(5.5) 



< 

< E 



E[i^ fe+n+1 (4)|z;] - K^vk+n+^zi) 

E[Kv k+n+1 (Z l n )\Z l n ] - Ki +1 v k+n+1 (Zi: 



\Kv k+n+1 (Z l n ) - Kv k+n +l(Zn 
By using the Lipschitz property of K stated in Lemma A. 4 we obtain 
~\Kv k+n+1 (Zi)-Kv k+n+1 (Zi)\ 



E 



(5.6) 



< | [Q] [v k+n+1 ] ^ + [Q] K+n+i], + C Vk+n+1 (E X + E 2 ) + E 3 \\\Zi - Z%- 

Then, recall that by construction of the quantized process, one has \Z l w S\\ = p l n {Z % w S^) . 
Hence we have the following crucial property: cr{Z^} C a{Z l n ,S' l ri }. By using the special 
structure of the PDMP {X(t)}, we also have a{Z t n ,S t n } C J^Tn, so tha t one has C 
a{Z^}. It now follows from the definition of K given in equation (4.1) that 



E[Kv k+n+1 (Zi)\Zi]-Ki +1 v k+n+1 (Z l n ) <E |F(z£,t*(4))-F(2*,t*(^)) 



+ E 



-as; 



n+1 



v k+n+l{Z % n+ i) 



-aS, 



™+iv k+n+1 (Z l n+1 )\ 



Z' 



From Lemma A. 3, we readily obtain 



E 



\F(Zlf(Zi))-F(Zlt*(Zi))\ 



Z'„ 



< E,\\Zl 



Z 



n\\pi 



(5.7) 



(5.8) 



and it is easy to show that 



E 



-aS, 



"■+lV k+n +l(Z l n+1 ) 



-aS. 



"+iv k+n+1 (Z l n+1 )\ 



Z' 



< [vk 



+n+l I 



~~ -^n+lllp + a Cv k+n+1 \\Sn+l — 5*7 



n+1 lip- 



(5.9) 
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Finally, recalling that C Vk < and combining equations (5.5)-(5.9) we obtain the 
expected result. □ 



We turn to the error made in replacing J by J^+i- Here we use the specific properties 
of quantization again, and the lemmas on indicator functions. 

Lemma 5.19 For all 1 < i < d, k € {1, . . . , N - 1}, n 6 {1, . . . , N - k}, and < rj < 
min {A(z)} ; one has 



max \j(Mv k+n+1 ,v k+n+ i)(Z l n ,t) - J l n+1 (Mv k+n+1 ,v k+n+1 )(Z l n ,t)\ 
teG(Zi) 

< | [Q] [v k+n+1 ] i C ^ + C ^(E l + E 2 + a [t*] ) + C C {E X + a [t*\ ) 
+ 2([c] 1 + [c] 2 [t*]) + i(^(l+[t*]) + C' e [t*])}||^-Z 



•~ 2Cf 
+ [v k+n+1 ] \\Z l n+l - Zl +1 \\ p + (— f- + C c )C x n. 



Proof: By definition of J, we have 

J(Mv k+n+1 ,v k+n+1 )(Z l n ,t) - j\ +1 (Mv k+n+ i,v k+n+1 )(Z l n ,t) 



(5.10) 



< Iv k+n+1 (Z l n ,t) -P n +lV k +n+l 

For the first term on the right hand side of equation ( |5.10 ), we proceed as for K in the 
preceding lemma 

Iv k+n+1 (Z l n ,t) - Z* +1 v fe+n+ i(Z;,t) < E \lv k+n+1 {Z l n ,t) - Iv k+n+l (Z % n ,t)\ 

+ \E[lv k+n+1 (Zi,t)\zi] - P n+1 v k+n+1 (Zl,t) 
On the one hand, it follows from Lemma |A.3| that 

Z 



max E 
' 1 



< 



\lv k+n+1 (Z l n ,t) - Iv k+n+1 (Z l n ,t)\ 

n [vk+n+ij^x + C Vk+n+1 [A] 1 (l + C\C t *)J + C Vk+n+1 C\[t*] ^\\Z l n - Z\\\ v . 

On the other hand, we use again the fact that a{ZJ n } C a{Zj l } to obtain 
^[lv k+n+ i(Z l n ,t)\Z l n ] - P n+l v k+n+l (Z l n ,t) 



< E 



+ E 



l {St +1 <tAt*(Zi)}\ 



*(zi)\\e aS ' k+1 Vk+n+i(Z l n+1 )-e a ^+iv k+n+1 (Z l n+1 ) 



zt 



v k+n+l (K+i) 1 



+1^1 - L {^+i<*At*(^)} 1 {5^ +1 <tAt*(^)} I 



Zl 



It remains to deal with the indicator function. Lemma 5.17| yields 
max |E[iufc +n+ i(Z£,t)|Z£] - P n+1 v k+n+ i(Z % n , t)\ 

teG(Z* n ) 

2 ~- 
— C«fc+n+iCA^ + (a + -)Cu fe+n+1 ||S^ +1 — 5^+i|| p 



[Vfc+n+l] \ \ Z n+l - Z n+l\\„ + 
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By using the same arguments and lemmas A.l and A. 3 we obtain similar results for the 
second term on the right hand side of equation (5.10), namely 



max E 

teG(Zi) 



HMv k+n+1 (Zi,t) - HMv k+n+1 (Zi,t)\ Z 
< { H i + M 2 [*1 + + Co) (Q- [A] 1 + (C A + a) [t*] ) } ||4 - Z%, 



and 



n\\P 



max \E[HMv k+n+1 (Z l n ,t)\zi] - H l n+1 Mv k+n+1 (Z l n , t) j 

< {<*[**] (C Vk+n+1 + C c ) + [c] 1+ [ c ] 2 [f] + S (C Wfc+B+1 + C C )}|K - Z 

2 i ~ 

+ ~(Cffe+n+i + Cc)\\S l n+ i — S'n+illp + CA(C„ fe+n+1 + C c )r/, 

showing the result. □ 
We now add up the preceding results to one obtains the following upper bound for T3. 
Lemma 5.20 A upper bound for T| is 

\\L d {Mv k+n+1 ,v k+n+1 )(Z l n ) - L^ +l (Mv k+n+ i,v k+n+1 )(Z l n )\\ p 

< \\Zi - frjJ [Q] [Vk+n+l]^ + ^{Ei +E 2 ) + j [Q] [u fc+ n + iL 

+2E 3 \vic c (Ex + a [f] ) + 2([c] 1 +[c] 2 [f] ) + i (1 + [i*] ) + C c [i*] 

2/.C/ 



+ll^ + i - -S^+illpi c f + ~l 2 !~ ^ 



=)} 



77 V a 
~ 2C7 

+ [^fc+n+l] ||-^n+l 



a 



5.3 Approximation of the value function 

Now we have computed the value functions on the control grid, we turn to the actual 
approximation of vq. As in the preceding section, we define the quantized approxi- 
mation of the underlying Markov chain (0 n ) starting from (xo,0), the actual starting 
point of the PDMP. Let (Z n , 5 f n )o< n <Ar-i be the quantized approximation of the sequence 
(Z n , S n )o< n <N~i- The quantization algorithm provides us with another series of finite 
grids r® C E x M + for all < n < N — 1 as well as weights for each point of the grids 
and transition probabilities from one grid to the next one. Let p n be the closest-neighbor 
projection from E x IR + onto r®. Then the quantization of n conditionally to Z$ = xq 
is defined by 

@n = (-^n> Sn) = Pn (Z n , Snj . 

We will also denote F% the projection of T® on E and the projection of T e on M + . We 
use yet again new quantized approximations of operators H , /, K, J and L d . 
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Definition 5.21 For v G L 2 , w defined on x G E, n G {0, . . . , N — 1} and z G 

consider 



H n+1 v(z, t) = E [e-CtA** to w (^ B| t A f ( f b)) ) l {§n+i ^ At , (fn)} 



Z n — z 



Wu; (^+i)l { s„ +1 < tAt *(^)} 



Z n = z 



K n+1 v(z) = E\F(Z n ,f(Z n )) + Q - aS ^v{Z n+l )\Z n = z 



J n+1 (v,w)(z,t) =E F(Z n ,i) + e- a5 "+ 1 w;(Z n+1 )l 



{S n+ l<iAt*(Z n )}l Z ™ - Z 



+ E 



-a(tA**(Z„)) 



l#(Z n , t A t*(Z n )))l { ^ +i > Mr( g n)} |Z n = Z 



l4 +1 (-t;,i//)(z) = min J n+ i(v,w)(z,t) A K n+ iw(z). 

ieG(z) 

With these discretized operators and the previous evaluation of the v k , we propose the 
following approximation scheme. 

Definition 5.22 Consider Vjy(z) = g{z) where z G Tfj and for k G {1, . . . , N} 

%_ x (z) = L d k (Mv k ,v k ){z), (5.11) 

where z G ^ k -\- 



See figure figure 5.1 for a graphical illustration of this numerical procedure. Therefore 
vq{Zq) will be an approximation of vq(Zq) = vq{xq). The derivation of the error bound 
for this scheme follows exactely the same lines as in the preceding section. Therefore we 
omit it and only state our main result. 

Theorem 5.23 For all < n < N — 1, suppose that A(z) for z G is such that 



'D*\\Z n -Z n \\ p + D 5 S n+1 -S n+1 



L> 3 



< min{A(z)}. 



Then we have 



\v n (Z n ) - v n (Z n )\\ p 

\\v n +l(Zn+l) ~ 

+3[u n +i] H^n+i — 2"n+i|L + 2C/||5 n+ i — S^+iJI + D^\\ A(Z n )\\ p 



< \\v n+ i(Z n+ i) - v n+ i(Z n+ i)\\ + max \v n+1 (y) - v n+ i(y)\ + D\\\Z n - Z n \\ p 



with 
Di 

Dl 
D 3 

D A 

D 5 



+2^/D3(D 4 \\Z n -Z n \\ p + DS\\S n+1 - S n+1 \\ p ), 
[v n ] + [Q] [v n+1 ] l C ^ + C ^{E 1 +E 2 ) 



a a 



+ 



+ 2E 3 } v{2([c] 1 + [c] 2 [t*]) +C c E 1 + a[t*](Q- + C c )}, 



V n +l\ 



c f + C Vn+1 C x +[c] 2 + (C c + C Vn+1 )(C\ + 
+ C C )C X , 



,2C f 



a 



,2C f 



2[t*](-^ + C c ), 



a 



a ' 
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Remark 5.24 By using the same arguments as in Remark 5.12\ it can be shown that 



\v n (Z n )—v n (Z n )\\p can be made arbitrarily small by an adequate choice of the discretization 
parameters. From a theoretical point of view, the error can be calculated by iterating the 



result of Theorem 5.23 , However, this result is not presented here because it would lead to 
an intricate expression. From a numerical point of view, a computer can easily estimate 
this error as shown in the example of section^ 

5.4 Step by step description of the algorithm 

Recall that the main objective of our algorithm is to compute the approximation vq{xq) of 
the value function of the impulse control problem vq(xq). The global recursive procedure 



is described on figure 5.1 



The calculation of vo(xq) is based on t he backward recursion given in Definition 5.22 and 
described in the first line of figure 5.1 It involves the operators Lj constructed with the 



quantized process © n starting from x$. This recursion is not self contained and requires 
previous evaluation of the functions Vj on the control set U. 



The lower part of figure 5.1, shows how to compute these functions vj at each point of the 



control grid U. This is the triangular backward recursion given in Definition 5.9 More 
precisely, define vm = g and set j < N and suppose that all the v\ for all I > j have already 
been computed everywhere on the control set U. One then computes Vj in the following 



way, following the j-th line of figure 5.1 counting from the bottom. One first iterates the 
operators L l ^ d and uses the quantized process n , to obtain Vj(y 1 ). Then one iterates the 
operators L 2 k ,d and uses the quantized process O^, to obtain Vj(y 2 ), and so on until the 
last point Vj(y u ). Thus one obtains Vj at all points of the control set U. 

5.5 Practical implementation 

The procedure defined above is the natural one to obtain convergence rates for our ap- 
proximations. However, in practice we proceed in a different order. 

The first step is to fix the computational horizon N. This point was discussed earlier. 
The second step is not the time discretization, but the computation of the quantized 
approximations of the sequences (On) an d (O n )- The quantization algorithm may be quite 
long to run. However, it must be pointed out that this quantization step only depends on 
the optimization procedure through the the control set U but it does not depend on the 
cost functions / and c. The sequence (O n ) is obtained in a straightforward way. As for 
the (O n ), if the control set is very small, it is possible to run as many sequences of grids as 
there are points in the control set. Otherwise, one can do with only one sequence of grids 
computed with the Markov chain (On) with a random starting point Zq = Zq uniformly 
distributed on the control set U. To derive the point-wise approximation error, one simply 
uses the finiteness of U and the definition of the L p norm. 



u i 

\v k (y l ) ~v k {y l )\ < u^\v k {y l ) -v k (y l )\- 

i=l 

< u l ^\\v k (Z^)-v k (Z^)\\ 



where u is the cardinal of U. Notice that the last term is bounded in Theorem 5.11 Hence, 
one really only needs two series of quantization grids. 
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Once the quantization grids are computed and stored, one computes the path-adapted 
time grids G(z) for all z in all the quantization grids, that is only a finite number of z. 
The step A(z) can usually be chosen constant equal to A, so that either one can store 
the whole time grids, or one only needs to store the values of A and t* (z) for all z in the 
quantization grids. 



Once these preliminary computations are done, one can finally compute the value 
function. This last step is comparatively faster. The only point left to discussion is how to 
choose the initializing function g. The most interesting starting function is the cost h of 
the no impulse strategy, because then the value function Vn has a natural interpretation. 
However, in general, one needs additional assumptions on Q to ensure that h is in L. 
Another problem, is that in general computing h is a difficult problem, especially as we 
need to know its value at many different points, as explained in Remark 5.1 To overcome 
these difficulties, one can choose g to be an upper bound of h, for instance, g = a~ 1 Cf. 
In the special cases where h can be explicitly computed, we advise to use h. 



6 Example 

Now we apply our procedure to a simple PDMP and present numerical results. This 
example is quite similar to example (54.29) in [6], we only added random jumps to obtain 
a non trivial Markov chain (Z n ,S n ). 



Set E = [0, 1[, and dE = {1}. The flow is defined on [0, 1] by <p(x,t) = x + vt for 
some positive v, the jump rate is defined on [0, 1] by A(x) = fix, with (3 > 0, and for 
all x G [0, 1], one sets Q(x, •) to be the uniform law on [0, 1/2]. Thus, the process moves 
with constant speed v towards 1, but the closer it gets to the boundary 1, the higher the 



probability to jump backwards on [0, 1/2]. Figure 6.1 shows two trajectories of this process 
for xq = 0, v = 1 and (3 = 3 and up to the 10-th jump. The running cost is defined on E 




0.5 



2 2.5 




D.5 



Figure 6.1: Two trajectories of the PDMP. 



by f(x) = 1 — x and the intervention cost is a constant cq. Therefore, the best performance 
is obtained when the process is close to the boundary 1. The control set U is the set of ^, 
< k < u — 1 for some fixed integer u. In this special case, the control grid is already a 
discretization of the whole state space of the process. Therefore one needs only one series 
of grids starting from the control points to obtain an approximation of the value function 
at each point of the control grid. 
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Figure 6.2: Approximated value function for N 




Figure 6.3: Approximated value function for N 




Figure 6.4: Approximated value function for N 
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We ran our algorithm for the parameters xq = 0, v = 1, ft = 3, cq = 0.08, the discount 
factor a = 2 and u = 50 points in the control grid and several values of the horizon N. 



For an horizon N = 5 (respectively, N = 10, N = 15) interventions or jumps, Figure 6.2 



(respectively, Figure 6.3, Figure 6.4) gives the approximated value function we obtained 
(computed at the 50 points of the control grid) for 50, 100 and 500 discretization points 
in each quantization grid and. As expected, the approximation gets smoother and lower 
as the number of points in the quantization grids increases. 



The theoretical errors corresponding to the horizon N = 5 (respectively, N = 10, 
N = 15) are given in Table 6.1 (respectively, Table 6.2, Table 6.3). The values of the error 



are fairly high and conservative, but it must be pointed out that on the one hand, they do 
decrease as the number of points in the quantization grids increase, as expected ; on the 
other hand their expressions are calculated and valid for a very wide and general class of 
PDMP's, hence when applied to a specific example, they cannot be very sharp. 



Number of points in the quantization grids 


\\v (Z ) -v (Z )\\ 2 


50 


4636 


100 


3700 


500 


2141 



Table 6.1: Theoretical errors for N = 5. 



Number of points in the quantization grids 


\\v (Z ) -v (Z )\\ 2 


50 

100 

500 


5.341-10 11 
4.501-10 11 
2.567-10 11 



Table 6.2: Theoretical errors for N = 10. 



Number of points in the quantization grids 


\\v (Z ) -v (Z )\\ 2 


50 

100 

500 


1.460-10 21 
1.288-10 21 
0.750-10 21 



Table 6.3: Theoretical errors for N = 15. 



Notice also that the approximated value function obtained for the horizon of N = 10 
jumps or interventions is much lower than that obtained for the horizon N = 5 jumps 
or interventions. This is natural as it is a minimization problem, and the more there are 
possible interventions the lower the value function is. This also suggests that the horizon 
N = 5 is not large enough to approximate the infinite horizon problem. Figure 6.5 gives 
the approximated value function we obtained (computed at the 50 points of the control 
grid) for respectively 500 points in the quantization grids and respective horizons of N = 5, 
iV = 10 and iV = 15 jumps or interventions. There is very little difference between the 
results for 7V = 10 and iV = 15, suggesting that it is enough to take an horizon of 10 jumps 
or intervention to approximate the infinite time optimization problem. 
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Figure 6.5: Approximated value function for N = 5, N = 10 and N = 15 for 500 points 
in the quantization grids. 



A Lipschitz continuity results 

A.l Lipschitz properties of the operators 

We start with preliminary results on operators M, H, F and /. 
Lemma A.l For any g defined on V, Mg € L. Moreover, 

N^Hi. i M 9] 2 <[c] 2 , [Mgl<[cl, C Mg <C c + C g . 

Proof: By using the fact that \Mg{x) — Mg(y)\ < sup zgU \ c(x, z) — c(y, z)\ and assump- 
tion 4.2 the result follows easily. □ 

Lemma A. 2 Let w£L. Then for all (x,y) 6 E 2 and (t,u) S M+, one has 
Hv(x,t) - Hv(y,u)\ < Di(v)\x-y\ + D 2 (v )\t - u\, 

where 

• ift < t*(x) andu < t*(y), 

D 1 (v)= [v] 1 + C v C t *[\] 1 , D 2 {v)= [v] 2 + C v (C x + a), 

• ift = t*(x) andu = t*(y), 

D i (*0 = H * + C v (C*. [A] 1 + (C x + a) [f]), D 2 (v) = 0, 

• otherwise, 

Di(v) = [v] 1 ^v] 2 [t*]+C v (C t .[\] 1 + (C x +a)[t*]), D 2 (v)=[v] 2 + C v (C x + a). 
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Proof: This is straightforward. 

Lemma A. 3 For all w G L, (x, y) G i? 2 and (t, u) G M+, one /tas 

F(x,t)-F(y,u)| < i ([/] x + C)C t . [A] x ) |x - y| + C f (\t - u\ V [i*] \x - y\) , 
/«;(*, t)-Iu;(y,u)| < i([Q] + C w [A] x (l + C A Q*)) |x - y\ 

+C w C x (\t-u\ V 

Proof: Suppose, without loss of generality, that t A t*(x) < u A t*(y). Then, one has 

tAt*(x) 



□ 



F{x,t)-F{y,u) 



< 



(is 



+ 



uM*(y) 



tAt*(x) 



-as—A(y,s) 



ds 



<{[f] 1 + C f Cr [A] x ) e- as d S |x - y| + C>|u A - t A t*(a;)|. 

From the fact that \u A - t A < |f — it| V [f ] \x - y\ we get the first inequality. 

By using similar arguments, it is easy to obtain the last result. □ 
Now we turn to the Lipschitz property of operator K. 



Lemma A. 4 For w G L and (x,y) G E 2 , one has 

C x 



Kw(x) -Kw(y)\<( [Q] H i ^ + [Q] M * + c » (^i + + ^ } 



Proof: This is a direct consequence of (4.1) and Lemmas A. 2 A. 3 



□ 



Finally, we study the the Lipschitz properties of operator J. 
Lemma A. 5 For all (v,w) G C 2 , x G E and (t,u) G , one has 

J(v,w)(x,t) - J(v,w)(x,u) < (Cf + C w C x + [v] 2 + C v (C x + aj\\t-u\ 



Proof: By using (4.1) and Lemmas A. 2 and A. 3 the result follows easily. 



□ 



Lemma A. 6 For all (v,w) G L 2 , (x,y) G E 2 and t>0, one has 

J(v,w)(x,t) - J(v,w)(y,t)\ <{[v] 1 + [v] 2 [t*] + C V E X + [Q] [w] 1 

+ C W E 2 + E 3 }\x - y\. 

where 

Ex = CrlXj^iCx + a)^*], 
1 + C x C t * 



Cx 
a 



a 



E 2 = C x [f] + [A]j- 
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□ 



Proof: Again, this is a direct consequence of (4.1) and Lemmas A,2| and A. 3 
Remark A. 7 It is easy to obtain that for (v,w) E C 2 , s E M+ and (x,y) E E 2 , 
inf J(t>, t) — inf J(u, t) < sup I J(u, w)(x, t) — J(v, w)(y, t)\ 

t>8 t>8 f> 

Lemma A. 8 Let (v,w) E L 2 . Then for all x E E and (s,t) E M+, 

inf J(v,w)(x,u) - inf J(v,w)(x, u) < [Cf + C^Ca + M, + C V (C\ + a))\t — s\. 

u>t u>s \ J 

Proof: Without loss of generality it can be assumed that s <t. Therefore, one has 

inf J(v, w)(x, u) — inf J(v, w)(x, u) = inf J(v, w)(x, u) — inf J(v, w)(x, u). (A.l) 

u>t u>s u>t u>s 

Remark that there exists s E [s A t*(x),t*(x)] such that inf J(w, g)(x,u) = J(w,g)(x,s). 

u>s 



Consequently, if s > t A t*(x) then one has inf J(v, w)(x, u) — inf J(y, w)(x, u) 

u>t u>s 

Now if s E [s A t* (x) , t A t*(x)[, then one has 

inf J(v, w)(x, u) — inf J(v, w){x, u) < J(v, w)(x, t) — J(v, w)(x, s). 

u>t u>s 



0. 



From Lemma A. 5, we obtain the following inequality 

inf J(v,w)(x,u) - inf J(v,w)(x,u) < (c f + C W C X +\v] + C V (C X + a))\t - s\. (A.2) 

u>t »' s »° \ - I 



u>t 



Combining equations (A.l), (A.2) and the fact that \t — s\ < \t — s\ the result follows. □ 



A.2 Lipschitz properties of the operator L 

Now we study the Lipschitz continnuity of our main operator 

Lemma A. 9 For all (v,w) E L 2 , x E E and t E [0, t*(x)) and u E M+, one has 

F{<j){x,t),u) = e at+K ( x ti[F{x,t + u)-F(x,u)\ 

Iw((P(x,t),u) = e at+A(x ^^Iw(x,t + u)- Iw(x,u)\ 

Hv((t>{x,t),u) = e at+A(x ^Hv(x,t + u). 

Proof: By using the semi-group property of <fi, we have A(cf>(x, t),u) = A(x, t + u) — A(x, t) 
for t + u < t*(x) and noting that t*(c/)(x,t)) = t*(x) — t for t < t*(x), a simple change of 
variable yields 



F(^(x,t),u) = e at+A( - x <V 



(t+u) At* (x) 



f(4>(x,s))ds, 



and we get the first equation. The other equalities can be obtained by using similar 
arguments. □ 

Lemma A. 10 For all (v,w) E L 2 , x E E and t E [0,t*(x)), 

L(v,w)((f)(x,t)) = e *"^*'*) {inf J(v,w){x,s) A Kw{x)} - F(x,t) - Iw(x,t) 
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Proof: For t G [0, t*(x)) and u G M+, we have from Lemma A.9| (4.1) and (4.1) 

J(v,w){</>(x,t),u) = e at+A{x ' t) [j{v,w)(x,t + u) -F(x,t) - Iw(x,t)], 
Kw(4>(x, t)) = e at+A( - x 'V [Kw(x) - F{x, t) - Iw(x, t)] . 



Consequently, from equation (4.3), it follows 

L(v,w)U(x,t)) = e a * +A(a: '*) { inf J{v,w)(x,t + u) A Kw(x)\ - F(x,t) - Iw(x,t) 
showing the result. 

Proposition A. 11 For all weL, Cw G L ; and 

M, < ^ + ^{[A] 1 ^(c c + ^) + ([ c ] 1 + [c] 2 [r] + c c E 1 )v([g][ 

+2E 3 + [ w ] i + { El + 2E 2 + [A] x Cr (1 + C A /a)}cA 

2 < e ( Q+c ^* \%C f +[c] 2 + 2C C (C X + a) + + C w [4C A + ^ + - 

[Cw] < {E 1 + E 2 }C w + ^^[w] 1 + E 3 +([c] 1 +[c] 2 [t*]+C c E 1 )v([Q}[ 



□ 



Proof: Let us denote Cw by g. We have for (x, y) G -E 12 and t G [0, i*(x) A t*(y)] 

g(cf>(x, t)) - g(J>(y, t))\ < e at+A ^ { \F(x, t) - F(y, t) | + \lw(x, t) - Iw(y, t) | } 
a *+ A (^)||inf J(Mw,w)(x,s) - inf J(Mw,w)(y,s) | V \Kw(x) - Kw(y)\\ 

inf J(Mw, w)(x, s) A Kw(x) \ - F(x, t) - Iw(x, t) 

s>t J 



+ e' 
+ 



e at+A(x,t) _ e at+A(y,t) 



It is easy to show that for x G E, t G [0, t*(x)], and weLwe have e °*+ A 0M) < e (a+c x )C t * ^ 
| mi s > t J(Mw,w)(x,s) AKw(x)\ - F(x,t) - Iw(x,t) < ±(Cf + C X C W ) + C C + C W and 

for (x,y) G E 2 and t G [0, t*(x) A t*(y)] e at +M^t) _ & at+A(y,t) < Q ( a +c x )C t * [\^C t *\x - y\. 
Consequently, by using Lemmas |A.3| and A. 4 and Remark A. 7 we get the first equation. 

For x G E and (s,t) G [0,t*(x)] 2 
g{(f>(x,s)) - g{(f>(x,t)) < e at+A ^\lw(x,s) - Iw(x,t)\ 

+ e at+A(x,t)(\ inf j( Mw ,w){x,u) - inf J(Mw,w)(x,u)\ + \F(x, s) - F(x,t)\\ 



+ 



e as+A(x,s) _ e at+A(x,t) 



\ inf J(Mw, w)(x, u) A Kw(x) > — F(x, s) — Iw(x, . 

I u>s J 



Note that for x G E, (s,t) G [0,t*(x)f 



-,as+A(a;,s) p at+A(x,t) 



<^ a+c ^*{C x + a)\t- 



s . 



Consequently, we have by using Lemmas A. 3 and A. 8 we obtain the second equation 
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The third inequality is straightforward and finally, for (x,y) £ E 2 we have 
g{x) — g(y) < I inf J(Mw, w)(x, s) — inf J(Mw, w)(y, s) I V \ Kw(x) — Kw(y)\. 

s>0 s>0 



By using Remark A. 7 and Lemma A. 4 we get the last equation. □ 



Corollary A. 12 For all < n < N , the value functions v n are in L. 



Proof: As vn = g is in L by assumption, a recursive application of Proposition A. 11 



yields the result. □ 
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